xavier nx 平台 EQOS LAN端口连接异常问题调试2

1. 前言

承接上篇
https://www.ebaina.com/articles/140000012768

2. 测试脚本

pe_test.sh

#!/bin/bash

### ENV DEFINITION ###

SERVERIP_ETH0="10.0.0.36"

DATE=`date "+%Y%m%d_%H%M%S"`

LOGFILE=/root/pe_log/petest_log_$DATE.txt

### FUNCTION ###

count=1

function print_usage () {

        echo "ex.)  ./pe_test.sh continue"  2>&1
        echo "ex.)  ./pe_test.sh poweroff"  2>&1
        echo "ex.)  ./pe_test.sh reboot"  2>&1
}

function test_and_check_init () {
        ITEM=$1;
        COMMAND=$2;
        EXPECTED=$3;

        RESULT=`eval $COMMAND`;

        if [[ $RESULT == *$EXPECTED* ]]; then
                echo "[$count] $ITEM [OK]"  2>&1 | tee -a $LOGFILE;
        else
                DATE=`date "+%Y%m%d_%H%M%S"`
                echo "[$count] $ITEM [NG] date=$DATE" 2>&1 | tee -a $LOGFILE;

                echo "arg num=$#"  2>&1 | tee -a $LOGFILE;
                echo "ITEM=$1"  2>&1 | tee -a $LOGFILE;
                echo "COMMAND=$2"  2>&1  | tee -a $LOGFILE;
                echo "EXPECTED=$3"  2>&1 | tee -a $LOGFILE;
                echo "RESULT=$RESULT"  2>&1 | tee -a $LOGFILE;

                # if check error at init, exit test
                exit;
        fi

        count=$(expr $count + 1);
        echo ""  2>&1 | tee -a $LOGFILE;
}

function test_and_check () {
        ITEM=$1;
        COMMAND=$2;
        EXPECTED=$3;

        RESULT=`eval $COMMAND`;

        if [[ $RESULT == *$EXPECTED* ]]; then
                echo "[$count] $ITEM [OK]"  2>&1 | tee -a $LOGFILE;
        else
                DATE=`date "+%Y%m%d_%H%M%S"`
                echo "[$count] $ITEM [NG] date=$DATE" 2>&1 | tee -a $LOGFILE;

                echo "arg num=$#"  2>&1 | tee -a $LOGFILE;
                echo "ITEM=$1"  2>&1 | tee -a $LOGFILE;
                echo "COMMAND=$2"  2>&1  | tee -a $LOGFILE;
                echo "EXPECTED=$3"  2>&1 | tee -a $LOGFILE;
                echo "RESULT=$RESULT"  2>&1 | tee -a $LOGFILE;
        fi

        count=$(expr $count + 1);
        echo ""  2>&1 | tee -a $LOGFILE;
}

if [ $# -ne 1 ]; then
  print_usage;
  exit 1
fi

if [ $1 != "continue" ] && [ $1 != "poweroff" ] && [ $1 != "reboot" ]; then
  print_usage;
  exit 1
fi

if [ ! -e /root/pe_log ];then
  mkdir /root/pe_log
fi

sleep 10

echo "### INIT CHECK ### " 2>&1 | tee -a $LOGFILE;

test_and_check_init "VERSION" "uname -a" "4.9.140"

test_and_check_init "EQOS" "dmesg | grep eqos" "eqos 2490000.ether_qos eth0: Link is Up"

test_and_check_init "ifconfig eth0" "ifconfig -a" "eth0"

test_and_check_init "usb storage" "lsusb -t" "Mass Storage, Driver=usb-storage, 480M"
#test_and_check_init "usb storage" "lsusb -t ; sleep 1" "Mass Storage, Driver=usb-storage, 5000M"

sleep 3

echo "###  EXECUTE CONTINUOUS TEST ###" 2>&1 | tee -a $LOGFILE;

testnum=0

while :
do

DATE=`date "+%Y%m%d_%H%M%S"`
echo "## TEST:$testnum START DATE=$DATE ##" 2>&1 | tee -a $LOGFILE;

test_and_check "PING from eth0" "ping -I eth0 $SERVERIP_ETH0 -c 4 -i 0.2" "4 packets transmitted, 4 received, 0% packet loss,"

sleep 1

testnum=$(expr $testnum + 1);

if [ $1 == "continue" ];then
  echo ""
elif [ $1 == "poweroff" ];then
  poweroff -f
elif [ $1 == "reboot" ];then
  reboot -f
else
  echo "Not continue/poweroff/reboot : Unexpected err" 2>&1 | tee -a $LOGFILE;
  exit
fi

done

3. 使用最新版本jetpack测试

用jetpack4.5.1再次构建了环境，并确认它可以复现
复现频率约为1/422
复现过程如下

1. 从https://developer.nvidia.com/jetpack-sdk-451-archive下载NX DEVELOPER KIT的SD卡镜像
2. 提取jetson-nx-jp451-sd-card-image.zip并写入sd-blob。img到micro SDCard
3.使用SD Boot, Setup和login引导NX DEVELOPER KIT
4. 从GUI关闭Wifi和蓝牙
5. 从界面中将eth0 IPv4设置为10.0.0.77/8
6. HostPC通过Gbit Ether HUB连接到EVK，如下所示
NX DEVELOPER KIT <=> NETGEAR:GS-105E <=> HostPC
10.0.0.77/8                              10.0.0.36/8

7. 在NX DEVELOPER KIT中部署以下shell脚本、服务并设置自动化测试
/root/pe_test.sh
/lib/systemd/system/pe_test.service

8. 在USB插槽中插入一个USB记忆棒，以防你有意中断测试
当USB闪存驱动器被拔出或问题重现时，重新启动测试停止
9. 从UART控制台重新启动并开始持续测试。

复现后的情况几乎与jetpack4.4.1相同
“Link is Up”事件不会从内核引发，IP地址也不会被分配
这不是定制板设计的问题

4. 开启CONFIG_EQOS_DISABLE_EEE

jetpack4.4.1/4.5.1中没有默认设置CONFIG_EQOS_DISABLE_EEE。
如果想禁用EEE, CONFIG_EQOS_DISABLE_EEE应该是Y

# uname -a
Linux contec-desktop 4.9.201-tegra #1 SMP PREEMPT Fri Feb 19 08:42:04 PST 2021 aarch64 aarch64 aarch64 GNU/Linux

# zcat /proc/config.gz | grep EQOS
# CONFIG_EQOS_APE_HWDEP is not set
CONFIG_EQOS=y
# CONFIG_EQOS_DISABLE_EEE is not set
# CONFIG_DISABLE_EQOS_CTRL_TRISTATE is not set

用CONFIG_EQOS_DISABLE_EEE=y测试了的定制板，但是问题仍然发生了
还没有尝试过EVK，应该先试用NVIDIA的原始内核映像

5. 测试标准

启用CONFIG_EQOS_DISABLE_EEE将有助于增强这个问题

因为测试条件不同

这是在板运行中连续的ifconfig down/up测试(没有重新启动)，PHY_AN连续发生了三天
但在避免重启测试中，即使使用CONFIG_EQOS_DISABLE_EEE=y，有时也没有发生PHY_AN

测试标准是连续重启两天，大约1500次开机，没有任何问题
与Jetson Nano相比，PHY_AN的错误率有明显的差异
而且，作为Gbit以太网的规范，自动协商不应该以如此高的错误率失败

6. 使用路由器连接

问题在下列配置中复现
中间加一个路由器

NX DEVELOPER KIT <=> NETGEAR:GS-105E <=> HostPC
10.0.0.77/8                              10.0.0.36/8

如果不想怀疑连接到路由器的其他节点和数据包的影响
认为最好是将节点直接连接在一起

7. 恢复正常状态

以下手动操作将恢复正常操作
注意，停机后需要等待大约5秒钟

ifconfig eth0 down

sleep 5

ifconfig eth0 up

这可能是因为自动协商序列将再次运行