Five shots were fired at 15:30 (06:30 GMT) on Saturday when the patrol boat crossed the de facto maritime border, South Korea's local news outlet reports.
North Korea disputes the maritime border in the Yellow Sea and has crossed it before.
Pyongyang called the shooting a "serious provocation".
The North's official KCNA news agency said the boat was conducting a "routine" operation.
In February last year South Korea said a patrol boat from the North had violated the line several times.
And ships from the two countries briefly exchanged fire near Yeonpyeong island last October.
The incident comes as the countries hold the second of two reunions for family members separated by the 1950-53 Korean War.
The reunions taking place this week are only the second round in the past five years.