In the IP packet header there is an 8-bit Protocol field (at offset 9) that holds the "next level protocol", with assigned numbers given in RFC 790. For example, TCP is 6. My question is, why is the IP layer aware of higher layers in the network stack?
Every "header" has some sort of "Next Protocol" identification field. This is necessary because on the wire, the data is nothing but a string of 1's and 0's. The receiving endpoint must have a way of interpreting what the next bits refer to.
If not for such a field which definitively indicates how to interpret the next set of 1's and 0's, there would be no way of determining if the next 32 bits were a TCP Sequence number or the IP Source Address (for instance). Both those fields are indeed 32 bits, so there is no way to programmatically determine what those 1's and 0's actually mean -- other than some sort of "Next Protocol" field.
The IPv4 protocol field allows the endpoint to determine what type of data is in the packet. The endpoint's IP layer uses the protocol field to determine which protocol to hand the packet off to.
Why is the IP layer aware of higher layers in the network stack?
An IP packet has a payload that must be passed to some other process. The IPv4 Protocol field (IPv6 Next Header field does the same thing) tells IP where to pass the payload of the IP packet.
The protocols in other layers have an equivalent type of field to know where to pass their payloads. For example, ethernet has the Ethertype field to tell it to which process (ARP, IPv4, IPX, IPV6, AppleTalk, etc.) to pass its payload, and transport protocols have addresses (ports) to tell them to which process to pass their payloads.
why is the IP layer aware of higher layers in the network stack?
Because IP is not the final destination of the packet payload. Each layer needs to know where to send its payload, but that does not mean that any layer really knows what is in the payload, or what the next layer actual is or does with the payload. As far as IP is concerned, it passes the packet payload to the process that registered with it at the protocol number, not that the payload is going to any specific transport protocol because IP knows nothing about transport protocols.
From Wikipedia:
Layer 3: Network layer
The network layer provides the functional and procedural means of transferring variable length data sequences (called datagrams) from one node to another connected to the same network. A network is a medium to which many nodes can be connected, on which every node has an address and which permits nodes connected to it to transfer messages to other nodes connected to it by merely providing the content of a message and the address of the destination node and letting the network find the way to deliver ("route") the message to the destination node. In addition to message routing, the network may (or may not) implement message delivery by splitting the message into several fragments, delivering each fragment by a separate route and reassembling the fragments, report delivery errors, etc.
Datagram delivery at the network layer is not guaranteed to be reliable. A number of layer-management protocols, a function defined in the management annex, ISO 7498/4, belong to the network layer. These include routing protocols, multicast group management, network-layer information and error, and network-layer address assignment. It is the function of the payload that makes these belong to the network layer, not the protocol that carries them.
Each layer supports the layers about it, that is the how the OSI Model is designed.
Protocols that can encapsulate multiple upper layer protocols need some way of indicating what that upper layer protocol is. Ethernet uses an Ethertype field, with 0x0800 denoting IP, and 0x0806 representing ARP. TCP uses well known port numbers, with e.g. 80 representing HTTP. HTTP uses the Content-Type: header field.
