1 目的

在centos7下使用qemu搭建arm模拟器。

参考博客：

centos下利用qemu搭建arm模拟器 - 寒水司天 - 博客园 (cnblogs.com)

从零使用qemu模拟器搭建arm运行环境_海枫的专栏-CSDN博客_qemu模拟arm

用Qemu模拟vexpress-a9 （一） --- 搭建Linux kernel调试环境 - 摩斯电码 - 博客园 (cnblogs.com)

2 环境

centos7版本：Linux version 3.10.0-693.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Tue Aug 22 21:09:27 UTC 2017

涉及到以下工具的使用：

arm交叉编译工具：arm-none-linux-gnueabi-gcc version 4.8.3

QEMU：qemu-2.11.0

uboot：u-boot-2018.01

Linux内核：linux-4.1.38

busybox：busybox-1.24.1

3 arm交叉工具安装

我下载的版本是：arm-2014.05-29-arm-none-linux-gnueabi-i686-pc-linux-gnu，直接解压缩之后就可以使用，不需要安装。

下载地址：

arm-none-linux-gnueabi-gcc 下载地址 - 代码天地 (codetd.com)

或者：Linaro Releases

安装步骤：

1、解压缩：

tar -vxf arm-2014.05-29-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2 

我是放在了/usr/local/arm路径下。

2、编辑vim /etc/profile

在文件末尾加上：

export PATH=$PATH:/usr/local/arm/arm-2014.05/bin

注意，上面的等号“=”是英文的，而不是中文的，否则会出现以下的错误。

-bash: export: `PATH＝/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/arm/arm-2014-05/bin': not a valid identifier

3、执行

source /etc/profile

4、测试

查看版本信息，这里遇到了一些问题，记录如下：

$ arm-none-linux-gnueabi-gcc -v
-bash: /usr/local/arm/arm-2014-05/bin/arm-none-linux-gnueabi-gcc: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

提示少了so格式不对，根据网友说法，安装glibc.i686之后解决了：

$ sudo yum install glibc.i686

查看版本信息：

$ arm-none-linux-gnueabi-gcc -v
... ...
Thread model: posix
gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-29) 

可以看出，安装的版本是：gcc version 4.8.3 20140320

4 安装QEMU

下载地址：

Index of / (qemu.org)，我下载的是2.11.0版本。

安装步骤：

1、解压缩：

tar -vxf qemu-2.11.0.tar.xz

2、配置编译选项：

./configure --enable-kvm --enable-debug --enable-vnc --enable-werror  --target-list=arm-softmmu --audio-drv-list=

configure脚本用于生成Makefile，其选项可以用./configure --help查看，这里使用到的选项含义如下：
--enable-kvm：编译KVM模块，使QEMU可以利用KVM来访问硬件提供的虚拟化服务。
--enable-vnc：启用VNC。
--enalbe-werror：编译时，将所有的警告当作错误处理。
--target-list：选择目标机器的架构。默认是将所有的架构都编译，但为了更快的完成编译，指定需要的架构即可。

这一步，出现了一些错误：

$ ./configure --enable-kvm --enable-debug --enable-vnc --enable-werror  --target-list=arm-softmmu --audio-drv-list=

ERROR: zlib check failed
       Make sure to have the zlib libs and headers installed.

缺少zlib开发包，需要安装好对应的版本：

$ yum install ghc-zlib-devel.x86_64

重新执行配置，报错：

$ ./configure --enable-kvm --enable-debug --enable-vnc --enable-werror  --target-list=arm-softmmu --audio-drv-list=

ERROR: glib-2.22 gthread-2.0 is required to compile QEMU

需要安装glib-2.22库：

$ yum install glib2-devel.x86_64 

重新执行配置，报错：

$ ./configure --enable-kvm --enable-debug --enable-vnc --enable-werror  --target-list=arm-softmmu --audio-drv-list=

ERROR: pixman >= 0.21.8 not present.
       Please install the pixman devel package.

需要安装pixman：

$ yum install pixman-devel.x86_64 

再次执行，终于成功了。

3、编译

$ make -j8

4、安装

$ make install
... ...
install -d -m 0755 "/usr/local/bin"
install -c -m 0755 qemu-system-arm  "/usr/local/bin"

5 安装uboot

下载地址：

Index of /pub/u-boot/ (denx.de)

我下载的是：u-boot-2018.01.tar.bz2

安装步骤：

1、解压缩：

tar jvxf u-boot-2018.01.tar.bz2

2、进入uboot源文件目录，执行：

export ARCH=arm
export CROSS_COMPILE=arm-none-linux-gnueabi-
make vexpress_ca9x4_defconfig
make

3、会提示gcc版本过低：

$ make
scripts/kconfig/conf  --silentoldconfig Kconfig
  CHK     include/config.h
  CFG     u-boot.cfg
  GEN     include/autoconf.mk
  GEN     include/autoconf.mk.dep
*** Your GCC is older than 6.0 and is not supported

可以选择一个高版本的GCC重新安装，或者修改uboot源文件的配置。

这里选择修改uboot源文件的配置：

$ vim arch/arm/config.mk 

注释掉了65行。

4、编译

直接执行make：

$ make

编译完成后，如果目录下生成 u-boot 文件，则说明编译成功。

5、在U-Boot源码目录下编写脚本 run.sh

qemu-system-arm \
                -M vexpress-a9 \
                    -nographic \
                    -m 512M \
                    -kernel u-boot

然后 chmod +x run.sh 增加文件执行权限。

6、最后执行 ./run.sh

可以看到启动了bootloader，但是会提示没有映像文件。

Wrong Image Format for bootm command
ERROR: can't get kernel image!

6 编译内核

下载地址：

Index of /pub/linux/kernel/v4.x/

我下载的是linux-4.1.38。

编译步骤：

1、从arch/arm/configs下找到需要编译内核的配置文件vexpress_defconfig，复制到源码根目录下。

$ cp arch/arm/configs/vexpress_defconfig ./

2、执行命令

export ARCH=arm
export CROSS_COMPILE=arm-none-linux-gnueabi-
make vexpress_defconfig
make zImage
make modules
make dtbs

3、编译后生成 在 arch/arm/boot 目录下生成 zImage 文件，则说明编译成功。

$ ls arch/arm/boot/
bootp/  compressed/  dts/  Image*  install.sh  Makefile  zImage*

这个时候可以直接用QEMU来执行测试一下内核是否正常：

$ qemu-system-arm -M vexpress-a9 -m 512M -kernel ./linux-4.1.38/arch/arm/boot/zImage -dtb ./linux-4.1.38/arch/arm/boot/dts/vexpress-v2p-ca9.dtb  -nographic -append "console=ttyAMA0"

这里简单介绍下qemu命令的参数：

-M vexpress-a9 模拟vexpress-a9单板，你可以使用-M ?参数来获取该qemu版本支持的所有单板
-m 512M 单板运行物理内存512M
-kernel ./linux-4.1.38/arch/arm/boot/zImage  告诉qemu单板运行内核镜像路径
-nographic 不使用图形化界面，只使用串口
-append "console=ttyAMA0" 内核启动参数，这里告诉内核vexpress单板运行，串口设备是那个tty。

看到内核的打印，说明内核正常，但是这时候没有文件系统，内核起不来。

此时只能通过杀死qemu-system-arm这个进程来退出。

ps -A | grep qemu-system-arm | awk '{print $1}' | xargs sudo kill -9

7 编译busybox

下载地址：

Index of /downloads (busybox.net)

我下载的是：busybox-1.24.1.tar.bz2。

操作步骤：

1、进入 Busybox 源文件目录下执行 make menuconfig。

$ cd busybox-1.24.1/
$ make menuconfig

选择上Build BusyBox as a static binary (no shared libs)。

2、执行make和make install

$ make
LINK    busybox_unstripped
Static linking against glibc, can't use --gc-sections
Trying libraries: crypt m
 Library crypt is not needed, excluding it
 Library m is needed, can't exclude it (yet)
Final link with: m
  DOC     busybox.pod
  DOC     BusyBox.txt
  DOC     busybox.1
  DOC     BusyBox.html

$ make install
--------------------------------------------------
You will probably need to make your busybox binary
setuid root to ensure all configured applets will
work properly.
--------------------------------------------------

_install 目录下生成 需要的文件 bin linuxrc sbin usr 。

$ ls _install/
bin/  linuxrc@  sbin/  usr/

该目录下的程序就是单板运行所需要的命令。

8 制作根文件系统

制作的根文件系统使用的是ext3文件格式。

先在Centos7主机环境下，形成目录结构，里面存放的文件和目录与单板上运行所需要的目录结构完全一样，然后再打包成镜像（在开发板看来就是SD卡），这个临时的目录结构称为根目录。

操作步骤：

1、创建rootfs目录（作为根目录），根文件系统内的文件全部放到这里。

sudo mkdir -p rootfs/proc/
sudo mkdir -p rootfs/sys/
sudo mkdir -p rootfs/tmp/
sudo mkdir -p rootfs/root/
sudo mkdir -p rootfs/var/
sudo mkdir -p rootfs/mnt/

sudo cp etc rootfs/ -arf

etc目录的下载地址：https://files.cnblogs.com/files/pengdonglin137/etc.tar.gz

2、拷贝busybox命令到根目录下

cp ../qemu/busybox/busybox-1.24.1/_install/* -r ./

3、从工具链中拷贝运行库到lib目录下

cp -arf /usr/local/arm/arm-2014-05/arm-none-linux-gnueabi/lib rootfs/
rm rootfs/lib/*.a
arm-none-linux-gnueabi-strip rootfs/lib/*

4、创建4个tty端终设备

sudo mknod rootfs/dev/tty1 c 4 1
sudo mknod rootfs/dev/tty2 c 4 2
sudo mknod rootfs/dev/tty3 c 4 3
sudo mknod rootfs/dev/tty4 c 4 4

5、生成32M大小的镜像

dd if=/dev/zero of=a9rootfs.ext3 bs=1M count=32

6、格式化成ext3文件系统

mkfs.ext3 a9rootfs.ext3

7、将文件拷贝到镜像中

mkdir tmpfs
mount -t ext3 a9rootfs.ext3 tmpfs/ -o loop
cp -r rootfs/*  tmpfs/
umount tmpfs

8、运行

qemu-system-arm -M vexpress-a9 -m 512M -kernel /home/qemu/kernel/linux-4.1.38/arch/arm/boot/zImage -dtb /home/qemu/kernel/linux-4.1.38/arch/arm/boot/dts/vexpress-v2p-ca9.dtb  -nographic -append "root=/dev/mmcblk0 console=ttyAMA0" -sd a9rootfs.ext3

完整的启动日志：

$ qemu-system-arm -M vexpress-a9 -m 512M -kernel /home/qemu/kernel/linux-4.1.38/arch/arm/boot/zImage -dtb /home/qemu/kernel/linux-4.1.38/arch/arm/boot/dts/vexpress-v2p-ca9.dtb  -nographic -append "root=/dev/mmcblk0 rw console=ttyAMA0" -sd a9rootfs.ext3   
WARNING: Image format was not specified for 'a9rootfs.ext3' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
Booting Linux on physical CPU 0x0
Initializing cgroup subsys cpuset
Linux version 4.1.38 (root@localhost.localdomain) (gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-29) ) #1 SMP Sun Dec 26 16:34:51 CST 2021
CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Machine model: V2P-CA9
Memory policy: Data cache writeback
CPU: All CPU(s) started in SVC mode.
PERCPU: Embedded 11 pages/cpu @9fbc0000 s16192 r8192 d20672 u45056
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 130048
Kernel command line: root=/dev/mmcblk0 rw console=ttyAMA0
log_buf_len individual max cpu contribution: 4096 bytes
log_buf_len total cpu_extra contributions: 12288 bytes
log_buf_len min size: 16384 bytes
log_buf_len: 32768 bytes
early log buf free: 14916(91%)
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 512652K/524288K available (4849K kernel code, 154K rwdata, 1376K rodata, 260K init, 148K bss, 11636K reserved, 0K cma-reserved)
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
    vmalloc : 0xa0800000 - 0xff000000   (1512 MB)
    lowmem  : 0x80000000 - 0xa0000000   ( 512 MB)
    modules : 0x7f000000 - 0x80000000   (  16 MB)
      .text : 0x80008000 - 0x8061c6a8   (6226 kB)
      .init : 0x8061d000 - 0x8065e000   ( 260 kB)
      .data : 0x8065e000 - 0x80684a00   ( 155 kB)
       .bss : 0x80687000 - 0x806ac1f8   ( 149 kB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
Hierarchical RCU implementation.
        Additional per-CPU info printed with stalls.
        RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.
RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
NR_IRQS:16 nr_irqs:16 16
GIC CPU mask not found - kernel will fail to boot.
GIC CPU mask not found - kernel will fail to boot.
L2C: platform modifies aux control register: 0x02020000 -> 0x02420000
L2C: DT/platform modifies aux control register: 0x02020000 -> 0x02420000
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 dynamic clock gating disabled, standby mode disabled
L2C-310 cache controller enabled, 8 ways, 128 kB
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x46420001
sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
clocksource arm,sp804: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
Console: colour dummy device 80x30
Calibrating delay loop... 400.58 BogoMIPS (lpj=2002944)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x60008280 - 0x600082d8
Brought up 1 CPUs
SMP: Total of 1 processors activated (400.58 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 0
clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
cpuidle: using governor ladder
cpuidle: using governor menu
of_amba_device_create(): amba_device_add() failed (-19) for /memory-controller@100e0000
of_amba_device_create(): amba_device_add() failed (-19) for /memory-controller@100e1000
of_amba_device_create(): amba_device_add() failed (-19) for /watchdog@100e5000
of_amba_device_create(): amba_device_add() failed (-19) for /smb/motherboard/iofpga@7,00000000/sysctl@01000
of_amba_device_create(): amba_device_add() failed (-19) for /smb/motherboard/iofpga@7,00000000/wdt@0f000
hw-breakpoint: debug architecture 0x4 unsupported.
Serial: AMBA PL011 UART driver
10009000.uart: ttyAMA0 at MMIO 0x10009000 (irq = 38, base_baud = 0) is a PL011 rev1
console [ttyAMA0] enabled
1000a000.uart: ttyAMA1 at MMIO 0x1000a000 (irq = 39, base_baud = 0) is a PL011 rev1
1000b000.uart: ttyAMA2 at MMIO 0x1000b000 (irq = 40, base_baud = 0) is a PL011 rev1
1000c000.uart: ttyAMA3 at MMIO 0x1000c000 (irq = 41, base_baud = 0) is a PL011 rev1
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Advanced Linux Sound Architecture Driver Initialized.
Switched to clocksource arm,sp804
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 2, 16384 bytes)
TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
hw perfevents: enabled with armv7_cortex_a9 PMU driver, 1 counters available
futex hash table entries: 1024 (order: 4, 65536 bytes)
squashfs: version 4.0 (2009/01/31) Phillip Lougher
jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
9p: Installing v9fs 9p2000 file system support
io scheduler noop registered (default)
clcd-pl11x 10020000.clcd: PL111 rev2 at 0x10020000
clcd-pl11x 10020000.clcd: /clcd@10020000 hardware, 1024x768@59 display
Console: switching to colour frame buffer device 128x48
clcd-pl11x 1001f000.clcd: PL111 rev2 at 0x1001f000
clcd-pl11x 1001f000.clcd: /smb/motherboard/iofpga@7,00000000/clcd@1f000 hardware, 640x480@59 display
40000000.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
40000000.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
Concatenating MTD devices:
(0): "40000000.flash"
(1): "40000000.flash"
into device "40000000.flash"
libphy: smsc911x-mdio: probed
smsc911x 4e000000.ethernet eth0: attached PHY driver [Generic PHY] (mii_bus:phy_addr=4e000000.etherne:01, irq=-1)
smsc911x 4e000000.ethernet eth0: MAC Address: 52:54:00:12:34:56
isp1760 4f000000.usb: bus width: 32, oc: digital
isp1760 4f000000.usb: NXP ISP1760 USB Host Controller
isp1760 4f000000.usb: new USB bus registered, assigned bus number 1
isp1760 4f000000.usb: Scratch test failed.
isp1760 4f000000.usb: can't setup: -19
isp1760 4f000000.usb: USB bus 1 deregistered
usbcore: registered new interface driver usb-storage
mousedev: PS/2 mouse device common for all mice
rtc-pl031 10017000.rtc: rtc core: registered pl031 as rtc0
mmci-pl18x 10005000.mmci: Got CD GPIO
mmci-pl18x 10005000.mmci: Got WP GPIO
mmci-pl18x 10005000.mmci: No vqmmc regulator found
mmci-pl18x 10005000.mmci: mmc0: PL181 manf 41 rev0 at 0x10005000 irq 34,35 (pio)
input: AT Raw Set 2 keyboard as /devices/platform/smb/smb:motherboard/smb:motherboard:iofpga@7,00000000/10006000.kmi/serio0/input/input0
ledtrig-cpu: registered to indicate activity on CPUs
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
mmc0: new SD card at address 4567
mmcblk0: mmc0:4567 QEMU! 32.0 MiB 
aaci-pl041 10004000.aaci: ARM AC'97 Interface PL041 rev0 at 0x10004000, irq 33
aaci-pl041 10004000.aaci: FIFO 512 entries
oprofile: using arm/armv7-ca9
NET: Registered protocol family 17
9pnet: Installing 9P2000 support
Registering SWP/SWPB emulation handler
rtc-pl031 10017000.rtc: setting system clock to 2021-12-26 13:20:20 UTC (1640524820)
ALSA device list:
  #0: ARM AC'97 Interface PL041 rev0 at 0x10004000, irq 33
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/smb/smb:motherboard/smb:motherboard:iofpga@7,00000000/10007000.kmi/serio1/input/input2
kjournald starting.  Commit interval 5 seconds
EXT3-fs (mmcblk0): using internal journal
EXT3-fs (mmcblk0): mounted filesystem with writeback data mode
VFS: Mounted root (ext3 filesystem) on device 179:0.
Freeing unused kernel memory: 260K (8061d000 - 8065e000)
random: nonblocking pool is initialized

Please press Enter to activate this console.