This study aimed to histologically characterize microscopic tumor invasion of the vasculo-biliary sheaths in perihilar cholangiocarcinoma so as to determine the primary tumor site and to clarify clinicopathologic differences according to the primary site. A retrospective analysis was conducted of 52 patients who underwent resection for perihilar cholangiocarcinoma involving both the hepatic hilus and the liver, verified histologically. Histologic features of the vasculo-biliary sheath invasion were evaluated by double staining with hematoxylin-eosin to assess general morphology and Victoria Blue to detect the elastic fibers of the vasculo-biliary sheaths. The perihilar cholangiocarcinomas were classified into extrahepatic-type (34 patients), featuring an extrahepatic component involving the liver, and intrahepatic-type (18 patients), featuring an intrahepatic component involving the hepatic hilus. Hemihepatectomy with extrahepatic bile duct resection was the most common surgical procedure. Tumor size (p = 0.002), pN classification (p = 0.005), and pM classification (p = 0.023) were significant independent prognostic factors. The primary site was not significantly associated with survival after resection (p = 0.214), as patients with extrahepatictype tumors had a cumulative 5-year survival rate of 32%, compared with 28% for patients with intrahepatic-type tumors. Double staining with hematoxylin-eosin and Victoria Blue permits histologic discrimination between tumors of extrahepatic and intrahepatic origin, and thereby determination of the primary tumor site in clinical cases of perihilar cholangiocarcinoma. Combining extrahepatic-type and intrahepatic-type tumors under the term perihilar cholangiocarcinoma is valid clinically, as these tumors show comparable surgical outcomes with similar clinical management.