In the US it takes an act of Congress to officially declare war, and (by international law) war can only be declared on an actual sovereign nation. Although the US recognized the State of Vietnam as a sovereign nation in 1950, it was separated into North and South Vietnam in 1954. Since the US did not recognize North Vietnam as a sovereign nation we could not (legally) declare war on them. Note that a similar thing took place at the start of the Korean War.
The President on the other hand, may initiate war-like actions if it is felt that we are under attack or have been attacked, or attack is imminent.
The 1st attack in the Gulf of Tonkin incident took place well within what most nations considered its territorial waters, ie 12 nm. The 1st attack took place at about 8-9 nautical miles from the coast of North Vietnam. (North Vietnam claimed the same 12 nm limit as Vietnam had before the split. but since the US did not consider North Vietnam a sovereign nation it could not have any territorial waters rights.) The USS Maddox fired first when the NVN torpedo boats closed to with 10,000 yds. Subsequently, in the press releases and the information presented to Congress, it was stated that North Vietnam had previously claimed only a 5 nm limit, and that the NVN torpedo boats fired first. The movement of the USS Maddox was a part of numerous incursions/raids by the South into North Vietnam waters, and was intended to be a show of force and a chance to gather intel.
The President initiated military action after the 2nd part of the Gulf of Tonkin incident, which did not occur. Note that it is quite possible that the President did not know that the 2nd attack was not real, as there is no evidence that he was ever told. However, McNamara, our intel community, and the Military knew that the 2nd attack did not occur. In his memoirs, McNamara acknowledged that he was aware of the falsity of the 2nd attack, but did not mention that he had informed the President of this.
For a Police Action to become an actual War Congress would have to pass a Declaration of War, but for whatever reason they were unwilling to do so.
All of the above information (including official documents) pertaining to the Golf of Tonkin incident was declassified in the early 2000s, and made public by the NSA at the Congressional hearings (2005?) on this matter and other matters pertaining to the US conduct in starting previous wars.