Understanding Physical and Virtual Memory

(The above image is created by Nandan Desai) 

In modern operating systems, applications always reference memory using virtual memory addresses. The above diagram illustrates what a Virtual Memory is! Virtual Memory is just a layer on top of RAM and some small portion of hard disk (known as pagefile or swap). Applications are only aware of the Virtual Memory and are not aware of the pagefile or the physical memory.

This Virtual Memory is divided into two parts.

• User-mode virtual address space ("user space")
• Kernel-mode virtual address space ("kernel space")

 (The above image is created by Nandan Desai)

The kernel and certain device drivers, which typically run in the privileged mode (a.k.a, kernel mode) of the processor, take up the Kernel-mode virtual address space. The rest of the processes, which typically run in user mode of the processor, take up the User-mode virtual address space.

The processes running in the kernel mode of the processor can access both user space and kernel space of the virtual memory.

Processes running in the user mode of the processor can only access the user space assigned to them.

On a 32-bit Windows machine, and with default configurations, the user space is 2GB and kernel space is 2GB. Each virtual address length is 32 bits and therefore, there are a total of 2^32 virtual addresses which can address a total virtual memory of 4 GB. Usually, the first half of the virtual addresses (0x00000000 to 0x7FFFFFFF) are for user space, and the next half (0x80000000 to 0xFFFFFFFF) are for the kernel space.

A peculiar difference between user space and kernel space is, the entire user space is private for each process! That means, when a process is loaded into the virtual memory, it sees that the entire user space (i.e., the address from 0x00000000 to 0x7FFFFFFF) is free for it to use! It can take up any address from the user space and it doesn't affect the other processes! This is the illusion that the Memory Manager of the Operating System creates for every user-mode process on the system! Every process thinks that it is free to take up any of the addresses in the range 0x00000000 to 0x7FFFFFFF. And that's why, two or more processes can have the same virtual address. But the physical address will obviously differ. We'll talk about how this mapping from virtual memory to physical memory is done later!

But as for the kernel space is concerned, it's shared amongst all the kernel-mode processes. If the kernel space ranges from the address 0x80000000 to 0xFFFFFFFF (as it is for the 32-bit Windows), then two kernel mode processes cannot have the same address in that range or otherwise, they will be overwriting each other.

The following diagram illustrates this clearly:

(The above image is taken from Microsoft Docs)

On 64bit Windows, user-mode and kernel-mode virtual address space is 128 TB each!

It's that huge because the total addressable size for a 64-bit machine is 2^64 bits (~2.305 Exabytes).

That means, each user mode process can take up any address between 0x000'00000000 to 0x7FFF'FFFFFFFF.

More on this Windows memory limits here: https://docs.microsoft.com/en-us/windows/win32/memory/memory-limits-for-windows-releases#memory-and-address-space-limits

Now, of course the user-mode or kernel-mode processes cannot consume 128 TB of memory because of hardware limitations. But the Memory Manager creates this illusion for the processes that they can use that much amount of memory addresses. That's why the memory that the processes see is called a Virtual Memory. Because it's not real!

So, to conclude, here is an illustration of how Notepad.exe and MyApp.exe's virtual memory layout may look like and how they are mapped to the Physical memory (and notice the virtual addresses and physical addresses of the two processes).

(The above image is taken from Microsoft Docs) 

Memory Management Internals

Today's modern systems use Paged Memory Management technique to manage the physical memory efficiently.

The physical memory is divided into contiguous blocks of fixed length called frames. And the virtual memory is divided into a contiguous blocks (of the same size as frames) called pages. Usually, pages and frames are 4KB in size. This size depends on the instruction set architecture and the processor type. The operating system selects one or more sizes from the page sizes supported by the architecture. The page sizes can vary from 4KB to sometimes 2MB or 4MB or even 1GB!

When a program is executed, it is loaded into the virtual memory (and has it's own virtual address space if it's a user-mode process, as explained earlier). As the program grabs pages of the virtual memory, these pages are mapped to the frames of the physical memory. This mapping is done by the Memory Manager of the Operating System.

The ultimate goal of a Memory Manager is to satisfy the memory requirements of all the processes running on the system. It does this by constantly moving frames from physical memory to the pagefile (on the disk) and vice versa. The processes are not aware of this and are in the illusion that they are just using pages of "memory" as required but behind the scenes, the Memory Manager keeps hustling and gives an impression to the processes that they are using the lightning-fast memory but, in reality, they might just have some of their pages residing on the disk! The moving of frames from physical memory to the pagefile is called Memory Paging or Swapping. Each Operating System (whether it's Windows or Linux or any other OS) has it's own algorithm of how the Memory Manager is implemented. But the overall concept of Memory Management remains the same across Operating Systems.

Page Table

The Memory Manager maintains a table where it stores the mapping-related data of pages and frames. This table is called as a Page table and each of the mapping entry in that table is called a Page Table Entry (PTE). Page table resides separately on the Physical Memory and the table content is directly managed by the Memory Manager of the OS.

Virtual address translation

Even the processor of the system sees the virtual address when it's executing an instruction of a process. If a process wants to read/write data on the "memory", the process specifies that to the processor through an instruction and the virtual address where that data has to be read/written.

When the processor goes to execute that instruction and tries to visit that virtual address, there is a special hardware component called Memory Management Unit (MMU) which intercepts this virtual address on-the-fly and replaces it with corresponding the physical address!

MMU refers to the Page table in the memory to do this virtual address translation. Remember that this Page table was created and is maintained by the Memory Manager of the Operating System. MMU only refers that page table to do the virtual address translation for the processor.

As the MMU has to go through the Page table every time it has to do the virtual address translation, there is another hardware component within the MMU called Translation Lookaside Buffer (TLB) which helps in speeding up this translation process.

TLB is just a small memory cache to store a few previously translated virtual address and their corresponding physical addresses. This cache is present only to speed up the translation process in the MMU.

So, here is how the entire virtual address translation process flows:

1. When a virtual address needs to be translated to a physical address, TLB is searched first.
2. If a match is found, which is known as a TLB hit, the physical address is returned and memory access can continue.
3. However, if there is no match, which is called a TLB miss, the MMU (or, in certain implementations, the operating system's TLB miss handler) will typically look up the address mapping in the page table to see whether a mapping exists, which is called a page walk.
4. If a mapping exists in the page table, it is written back to the TLB, and the faulting instruction is restarted. And now the MMU will obviously find the mapping in TLB and the process carries on normally.
5. If a mapping exists in the page table, but the Page Table Entry (PTE) is marked as "moved out" (i.e., the frame corresponding to the page has been moved out of the physical memory and to the disk), then the MMU raises a Page fault exception. This page fault exception will be handled by the Memory Manager of the Operating system. The Memory Manager needs to bring back that frame to the Physical memory, update the Page table, and then restart the instruction that had a Page fault.
6. If the mapping does not exist in the Page table, then the MMU will call the Operating System to handle this case. The OS will send a Segmentation Fault signal to the offending program which usually leads to a program crash.

Page table implementations differ on different systems. Some systems used to maintain a single global Page table while some others maintain a Page table for each process. There are also some implementations where the Page tables are stored in Virtual Memory. That means, the Page tables need to be paged themselves. And we also have hierarchical page tables. The complexities increase as we go deeper into the implementation details of various systems. But the concept of memory management described in these notes remains the same.

It's important to note that the Paging needs to be enabled by the kernel during the system boot process. On Intel processors, Paging is enabled by setting the PG flag of a control register named cr0. When PG = 0, virtual addresses are interpreted as physical addresses.

The Memory Manager initializes the Page Table in the Physical memory and then enables Paging by setting the PG flag of cr0 control register. From that point onwards, MMU starts considering all addresses as virtual addresses and does the translation as described previously.

Direct Memory Access (DMA)

There are certain peripherals (such as Thunderbolt ports) on some PCs that allow the devices to access the Physical memory directly. The data from these devices doesn't flow through the CPU. And the MMU doesn't touch these DMA instructions. DMA devices can directly work with the Physical memory to speed up the data transfer. But how much or what part of Physical memory is allowed to be accessed by such devices depends on the system architecture. This direct access to the Physical memory is called Direct Memory Access (DMA). There were/are certain vulnerabilities in DMA and are exploited by the attackers to bypass OS security checks and gain access to part or whole of the Physical Memory and steal data or install a malware on the Physical memory. Such attacks are called "DMA attacks".

 

References:

https://static.lwn.net/images/pdf/LDD3/ch15.pdf

https://doc.lagout.org/operating%20system%20/linux/Understanding%20Linux%20Kernel.pdf

https://tldp.org/LDP/tlk/mm/memory.html

https://www.kernel.org/doc/html/latest/admin-guide/mm/concepts.html

https://en.wikipedia.org/wiki/Memory_management_unit

https://techcommunity.microsoft.com/t5/ask-the-performance-team/pages-and-page-tables-8211-an-overview/ba-p/373113

https://techcommunity.microsoft.com/t5/ask-the-performance-team/memory-management-101/ba-p/372316

https://www.cs.cornell.edu/courses/cs4410/2015su/lectures/lec14-pagetables.html

 

Monitoring Android App's Network Activity

Starting an emulator outside Android Studio

nandan@nandan-pc:~/Android/Sdk/emulator$ ./emulator -list-avds
Pixel_2_API_28
Pixel_2_API_R
Pixel_2_XL_API_29
Pixel_XL_API_25
nandan@nandan-pc:

 ~/Android/Sdk/emulator$ ./emulator -avd Pixel_XL_API_25  -writable-system

Connecting the emulator to mitmproxy

1. If you are connected to both wifi and mobile data in your emulator and wifi is showing 'no internet', then first turn off the wifi.
2. Next, run your mitmproxy. Preferably mitmweb.
3. Now, to connect to mitmproxy, we need to change our proxy settings. To do that, follow this article.
4. Now, open Chrome in your emulator and visit http://mitm.it. If you can see that certificate install page, then everything is fine. But don't install certificate there. Since Android 7, apps can opt-out of User added CAs and hence you can't see the HTTPS traffic of many of the apps if you install the certificate from mitm.it. We need to make the mitmproxy certificate as a system-added certificate authority.
5. To do that, follow this article.
6. Now, while following that article, I encountered several problems. My adb was not able to run as root and my adb root command was failing. That's when I read that emulators are of different types. We need to have emulators labelled as 'Google API' to get many debug options. On the other hand, the emulators which are labelled as 'Google Play' won't have many options as they are "production builds" of the Android OS. You can see this label while downloading a new emulator in AVD Manager in Android Studio.
   
7. The emulator that ran flawlessly was Pixel_XL_API_25. Some other emulators even with 'Google API' label were encountering some kind of errors while following the article mentioned in point 5.
8. There is also a case where the app developer can get clever and add specific network configurations to trust certain standard certificates only. This way, even if you have made mitmproxy as a system certificate in the above steps, you'll still not be able to view that app's HTTPS traffic. To bypass this, you can change that configuration by decompiling the app, add your own configurations and then recompiling the apk. Although I have not tried this personally but there is a repo here that claims to do this for you.

EDIT: You will need to specify the mitmproxy IP address and port in the emulator settings. There, just enter 127.0.0.1 as IP and whatever the port mitmproxy is running on.

How do I download APKs and install them in emulator?

1. I prefer to use Raccoon desktop app to download the APK file and then drag-n-drop the apk into the emulator to install it.
2. But, recently, Android has introduced App Bundles. Which means the APK file is split into multiple files. And we can't just drag and drop the files to install the app. To install such split APK files, we will use Split APKs Installer (SAI).
3. Now just drag and drop the whole folder that contains the split apks into the emulator.
4. You will find that folder in Downloads folder in emulator.
5. Open SAI and navigate to the split APKs folder and select the APKs of the app you want to install. Make sure you select the proper APKs (like when I selected armeabi apks, the installation was failing for my particular case).
6. That's it! Now the installation is complete and you can run the app.

Tips on monitoring the traffic of an app.

Now that you have followed the above explanation, here is a tip. Install NetGuard in the emulator. It will help you block all the other apps in the emulator from accessing the internet and you can single out your target app to access the internet. That will be very easy in mitmproxy web interface to study the traffic rather than filtering the traffic in mitmproxy itself.

Summary on starting the Android app network monitoring process.

After all the (above mentioned) setup is ready,

1. Turn on the mitmweb. It will serve proxy server on port 8080 and web interface on 8081
2. Turn on your emulator with the following command

   ./emulator -avd Pixel_XL_API_25 -writable-system
   

3. Enjoy the network analysis of your target android app!

Problems faced during app analysis

An app used to detect whether it's running on a rooted device or an emulator.

I used JADX to get the logic of how the app was checking the root and the emulator. But JADX can only be used for viewing and analyzing a decompiled APK. It can't be used for editing and recompiling it.

To edit the APK and recompile it, we need to use apktool jar file.

java -jar apktool_2.4.1.jar d target-app.apk

Decompiled version will be in smali. And the file names are obfuscated. So, by referring to JADX, we can find the right file and the right method to modify in smali. After the modification, we need to build back the APK.

java -jar apktool_2.4.1.jar b target-app

The modified APK can be found in the dist/ folder inside the decompiled folder. Now, before we can install this modified APK in our emulator, we need to sign it.

Create a new keystore and add some dummy data in it.

keytool -genkey -v -keystore target.keystore -alias alias_name -keyalg RSA -keysize 2048 -validity 10000

Then sign the modified APK with the newly generated keystore. For that, install apksigner (sudo apt install apksigner)

apksigner sign --ks target.keystore target-app.apk

Verify the signature once again:

apksigner verify --verbose target-app.apk

Done! Now the modified app will run smoothly in the emulator!

Also, if there are any problems while installing it, try installing the app using adb into the emulator to get a detailed log.

adb -s emulator-5554 install target-app.apk

Routing mitmproxy traffic through Tor

1. Install Tor on your system (sudo apt install tor).
2. Start Tor with it's HTTPTunnel proxy at any port (I will use 9060).

nandan@nandan-pc:~$ tor --HTTPTunnelPort 9060

1. Start mitmproxy in upstream mode

./mitmweb --mode upstream:http://127.0.0.1:9060 --set termlog_verbosity=debug

That's it! Now the emulator's traffic will go through mitmproxy and then through Tor.