am65xx: Remove dependence on DEVMEM entry