If you're talking about a telephone key pad or if it follows a number say "pound" or "hash" (if you are using US or UK English respectively) as in "enter your password followed by the # sign" or "a 5# bag of sugar." If it's in a tweet say "hashtag."