What is new in Windows 2008 R2 Server Core
http://technet.microsoft.com/en-us/library/dd883268(WS.10).aspx

The supported operating system list does not mention Windows Server 2008 R2 Server Core!
So officially it is not supported to install Exchange 2010 on Windows Server 2008 R2 Server Core!

The following table lists the supported operating systems for Exchange 2010
http://technet.microsoft.com/en-us/library/aa996719.aspx

Operating system on a computer that has a 64-bit processor One of the following:

• 64-bit edition of Windows Server 2008 Standard with Service Pack 2 (SP2)
• 64-bit edition of Windows Server 2008 Enterprise with SP2
• 64-bit edition of Windows Server 2008 R2 Standard
• 64-bit edition of Windows Server 2008 R2 Enterprise

But because we are very curious whether not supported also means does not work we want to try to install it anyway and see if it works :)

Install .NET Framework and Powershell onto the Server Core
Now we can install the prerequisites for Exchange on Server Core
http://technet.microsoft.com/en-us/library/bb691354.aspx#WS08R2

Let’s try to add the necessary windows components for the default Exchange 2010 installation (Hub/Cas/Mailbox)

The install fails because RSAT-ADDS is not found so let’s delete that from the list and try again…

The install fails because Web-Legacy-Mgmt-Console is not found so let’s delete that from the list and try again…

The install fails because RSAT-Web-Server is not found so let’s delete that from the list and try again…

The install fails because RPC-Over-HTTP-Proxy is not found so let’s delete that from the list and try again…
(Note that CAS Server will probably not work because of this error)

So the install of System Components worked when we left out all Management Console components and the RPC Over HTTP Proxy
So far so good (partially anyway)

Now let’s see if we can run setup…

Setup runs but the windows is blank so probably needs components not available on Server Core

So let’s try to do it via the Exchange 2010 setup.com command line

Prerequisites were fine but the install fails :(
The following error was generated when "$error.Clear(); install-ContentIndexingService -HubOnlyFilterRegistration" was run: "Object reference not set to an instance of an object.".

So there seems to be a problem with the indexing service not interacting with setup correctly

Setup failed but the HUB role was installed but none functional, it is listed in the console (the CORE server)

Trying to uninstall the HUB role also generated an error

Uninstall did not work and generated the very same error at intall time so I could not uninstall the Hub role :(
I had to forcibly remove the Server from the Active Directory Configuration partition with ADSIEDIT.MSC

So I rebuild the Server Core completely and tried to install the Mailbox role only…

Same error as installing Hub role, ContentIndexingService failed :(

Conclusion:
Installing Exchange 2010 on Windows 2008 R2 Server Core is not supported and does not work because install does not succeed
Most likely because Microsoft did not test any of the scenario’s on Server Core and also because there is only a subset available of some components used by Exchange 2010.