Grayback Brush Riders began in Williams, Oregon in 1948. Williams is a small town in the Siskiyou Mountains, southwest of Medford and at the "end of the road". Grayback Mountain is the namesake peak that can be seen from all over Williams. At the time the club began, most people who lived in Williams were ranchers, farmers, loggers, and miners. Cleo Kincaid, then the postmistress, Mabel Patterson, Waldine Terry, Pearl Hart, John Lathrop, the Pattersons, the Boyces, and Percy and Connie Sowell were some of the original members. The club first built an arena on a ranch near the Hartley cemetery. The Sowells bought the five acre property on Davidson Road where the arena and clubhouse are now, and donated it to the club.