I cant figure out how to remove it. Does anybody know? Thanks.
Those green circles and direction markings only show up in the symbol view and no longer show up in the schematic when the library is used. Here is an example.
This is the symbol view of a part in a library:
And this is the symbol when it is actually implemented in a schematic:
I assume you don't care about these markings in the design view of your part symbol as the users of your library will not see these markings.