This cast does not seem correct, since we should be writing to a pointer,
not a ulong.
This fixes the following warning on nds32:
-ns16550.c:49: warning: passing argument 2 of 'writeb' makes pointer from integer without a cast
Signed-off-by: Simon Glass <sjg@chromium.org>