Couldn't find anything on it. The main military action in that period within California wasn't Civil War related, but rather was related to the Mexican War from 1848-1849, so I thought it might have been related to that. But not too sure there was even an area called Baldwin Hills back then; the modern "black Beverly Hills" wasn't developed until about the 1920s according to its Wikipedia page. Then again, I couldn't find for whom Baldwin Hills was named after, so it's still possible that it was an undeveloped location before that time.
I heard this about 15 years ago. I was there over the weekend for a(nother) wedding reception. I told my wife I would get a metal detector and search for bullets and rifles. She laughed.